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The present invention relates to a user interface, and in particular to a user interface with 
a gesture based user interaction, and devices including such a user interface, and 
5 computer program code and computer program products providing such an interface. 

The present invention addresses problems with user interlaces and in particular user 
interfaces for devices with small displays, such as mobile computing devices, PDAs, and 
cellular communications devices, such as mobile telephones and smart phones and 

10 similar. However, the benefits of the invention are not limited to such devices and the 
invention can also be of utility in connection with desk top, lap top or note book 
computing devices and for devices witii large displays, such as data boards. Further the 
invention is not limited to utility with electronic devices whose primary function is 
computing, and can be utilised with any electronic device having a display via which a 

1 5 dialogue can be carried out with a user. 

A difficulty with designing graphical interfaces for small displays, such as touch screen 
displays, is that a regular text document has to be divided into very small pages, making Tl 

comprehension awkward. An additional problem is control elements take up precious H 

*> 

20 display area, making the view of a document ever smaller. One approach is to reduce liie ^ 

> 

size or nuniber of control elements, so as to free up usable display area. However this 

effects the usability of an interface. Hence a problem is to maintain a reasonable sized ^ 

interface without affecting its usability. ITl 

O 

o 

25 The difficulty in constractmg good solutions to interaction, particularly for handheld and ^ 
portable devices with small graphical displays, has spawned much interest from 
researchers specializing in multi modal and tangible forms of interaction. Some of the 
previous approaches to command and text input will be reviewed to set the benefits of the 
present invention in suitable perspective. 

30 

Many proposed solutions to the handheld command and/or text input problem fail to 
appreciate the trae obstacles of preserving portability and compactness, ease and 
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convenience of interaction and the deft conservation of scimi real estate. In order to 
illustrate the problem of text input for handheld devices, some previous approaches will 
be discussed 

5 Plug-in keyboards, or the laser projected variety, such as the virtual laser keyboard 
provided under the name iBE, would seem to offer a solution to the problem of easily 
entering text on small devices. However, this approach reduces the portability of a 
device and requires a user to carry ancillary equipment. The integration of a full size 
keyboard into a device design compromises the necessary limit on size and ergonomics of 
10 use, not to mention the portability of the device, as a flat surface is required to use the 
keyboard. 

A different approach is the choided keyboard, more usefiilly implemented for handheld 
devices as a device held in the hand. However, there is a significant learning overhead 
1 5 due to the user having to learn key combinations to select each letter or number. This 
approach does provide high one handed text input rates o£ for example, more than 50 
words per minute. However, with current hnplementations the need to hold a chorded 
keyboard in one hand, does affect the ergonomics of interaction. A modified approach 
would be to integrate the keyboard into the device itself 

20 

Similar to the chorded keyboard is the T9 predictive text found on many mobile phones. 
Entering a series of characters using keys generates a Ust of possible words. This 
approach does pose difficulties if the intended word is not found in the dictionary or the 
intended word is at the bottom of the list of suggestions. 

25 

Clip on keyboards may appear to provide a usable tjext entry fecility for small devices, at 
least on physical grounds. However, they do add bulk, and flius adversely aflfect the 
trade-off between size, portability and practicality. An alternative to the clip on is the 
overlay keyboard. Thougji these do not increase tiie size of the device, they do have 
30 usability implications. The overlay keyboard is essentially no different to a soft keyboard 
(discussed below), and can be a sticker that permanently renders the utility of a portion of 
the display for text input only, thereby restricting the use of an already limited resource. 
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The soft keyboard is not substantially different fiom the clip-on keyboard, except that it is 
implemented as a graphical panel of buttons on the display rather than a physical sticker 
over file display. The soft keyboard has the added hindrance of consuming screen display 
area, as does the overlay approach. However, as the soft keyboard is temporary, it does 
5 permit the user to fi-ee-up display area when required. While tiie soft keyboard approach 
appears to be a commonly accepted solution, it is a solution that is greedy in temis of 
screen area. 

Another approach based on the standard keyboard is one that uses a static soft keyboard 
10 placed in the background of the display text. Aletter is selected by tapping the 

appropriate region in the background This solution permits manual input and does 

preserve some screen real estate. However, the number of available controls and hence . 

redundancy is limited due to the necessary larger size of the controls, required to make 

the keys legible through the inputted text. Hiis limit on the number of controls 
15 necessitates an awkward need to explicitly switch modes for numbers, punctuation and 

other lesser used keys. Another drawback is the slight overhead in becoming accustomed 

to the novel layout. 

Attempts have been made to improve the soft kej^^oard approach^ but these attempts are 
20 still subject to the drawbacks already describe wife this approach. Further, they are 
subject to a learning overhead imposed by remodelling the keyboard layout In a 
Unistroke keyboard, all letters are equidistant, thus eliminating excessive key homing 
distances. A Metropolis keyboard is another optimised soft keyboard layout, which has 
been statistically optimised for single finger input Efficiency is improved by placing 
25 firequentiyused keys near flie centre of the ke;^oard. While both approaches can be 
effective, but both impose a learning overhead due to a new keyboard layout. The user 
must expend considerable effort to become familiar with the keyboard for relatively slim 
rewards, not to mention tiie overhead inherent with soft keyboards, such as the 
consmnption of screen real estate. 

30 

Handwriting recognition was for some time flie focus of PDA text input solutions. 
However, evaluation has revealed that gesture recognition for text input is balky and . 
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slower, some 2Swpm at best, than that of other less sophisticated approaches^ such as the 
soft keyboard A problem with handwriting, and similar approaches using 2D gesture 
interaction, such as Graffiti, is one of leamability, slow intemction and skill acquisition. 
A problem with handwritten input is the need, and time expended, to write each letter of a 
word. Irrespective of whether this is consecutively, or all at once, the user miist still write 
the whole tiling out In contrast a keyboard based solution requires merely the pressing of 
a button. 

In addition to this difficulty, as with the standard soft keyboard, text input requires the use 
of a stylus, thus occiq)ying the user's free hand the need to hold the PDA or device) 
when entering text. The learning curve of this approach is steep due to the need to learn 
an alphabet of gestures and the savmg in real estate is not so apparent^ since some 
approaches require a large input panel. 

Another, less well known, solutions to the problems of text entry for small devices is the 
use of a mitten. Sensors in the hand units measure the finger movements, while a smart 
system determines appropriate keystrokes. While this approach is an intriguing solution, 
a problem with it is the need to cany around a mitten that is nearly as big as the device 
itself. Further, a mitten may not be appealing to the user and the sensors on these devices 
can be bulky affecting freedom of movement 

A frirther approach is known as Dynamic dialogues, which, when applied to limited 
display size, provides a data ^try interface which incorporates language modelling. The 
user selects strings of letters as they progress across the screen. Lettei^ witii a higher 
probability of being found in a word are positioned close to the centre line. Allhough the 
dynaniic dialogue approach makes use of 2D gestures, these are supported by affordance 
mechanisms and they have been kept simple for standard interaction, making them readily 
leamable. Users can achieve input mtos of between 20-34 words per minute, which is 
acceptable \dien compared witii typical one-fingw keyboard touch screen typing of 20 -30 
words per minute. However, the input panel for text entry consumes around 65% of the 
display, leaving as little as 15% remaining for the text field. The approach does not 
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improve on the constraints of limited display area or on text input rates. What it do^ do 
is require the user to become &miliar with a new technique for little benefit 

The present invention therefore aims to provide an improved user interface for entering 
5 commands and/or text into a device. The invention addresses some of the above 

mentioned^ and other problems, as will become appar^t firom the following description. 
The invention applies superimposed animated graphical layering, (sometimes referred to 
herein as visual overloading) combined with gestural interaction to produce an 
overloaded user interface. This approach is particularly applicable to touch screen text 
10 input, especially for devices with limited display real estate, but is not limited to that 
application nor to touch screen display devices. 

According to a first aspect of the present invention, &ere is provided a usct interface for a 
display of an electronic device, the user intof ace inchiding a background layer and at 
15 least a first control element overlaid on the back ground layer. The control element has a 
plurality of functions associated with it. Each of said fimctions can be selected, invoked 
or executed by making a 2D gesture associated one the fimctions in a region of the user 
inter&ce associated with the control element. . The control elem^t can be transparent. 

20 In this way the amount of the displaying available for displaying information is increased, 
without reducing fimctionality as a user can easily select and execute a fimction or 
operation by simply making the appropriate 2D gesture over the control element. 

The background layer can display an interfiice, work context or dialogue for an 
25 application with which the user is interacting via the inter&ce. For example, the 
background layer can display text, a menu, any of the elements of a WIMP based 
interface, buttons, control elements, and similar, and any combination of the aforesaid. 

The control element can be animated In particular, the shape, size, form, colour, motion 
30 or appearance of the control element can be animated or otherwise varied witii time. An 
animated control element helps a user to distinguish between the control element and 
background while still rendering tihe background easily viewable and readable by the user. 
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The control element can also move over a region or the whole of the background 
Preferably the control element continuously moves over and repeats a particular patii, 
track or trace. The path track or trace may be curved 

5 The control element can be opaque. The control element can be at least partially 

transparent Parts of the control elem^t can be opaque and parts of the control element 
can be partially or wholly transparent Parts of the control element can be partially 
transparent and parts of the control element can be wholly transparent The whole of the 
control element can be transparent at least to some degree. Alpha blending can be used to 
1 0 provide a transparent part of a control element or control element 

The control element can be any visually distinguishable entity or indicia. For example, 
the control element can be a character, letter, numeral, shape, symbol or similar of any 
language, or combination or string thereof The control element can be an icon, picture, 
1 5 button, menu, tile, title, dialogue box, word or similar, and any combination thereof. 

The 2D gesture can be a straight line or a curved line, or combination of curved and/or 
straight portions. The 2D gesture can be a character, letter, numeral, sh^, symbol or 
similar of any language, or combination or string thereof. The 2D gesture can be 
20 continuous or can have discrete parts. 

The control element can be a word Different characters or groups of characters of the 
word can be animated separately. The word can be a polysyllabic word and each 
individual syllable can be animated 

25 

The control element can be a button or menu title. The button or menu title can bear an 
indicia, such as a symbol, word, icon or similar (as mentioned above) indicating a menu 
or group of functions or operations associated witii the button and making the 2D gesture 
can select of execute a function from the menu or group. 

30 

A help function can be associated with the control element Making a help 2D gesture 
can cause help information relating to the fimctions associated with the control element to 
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be displayed in the user int^:&ce. The infonnation can be displayed adjacent and/or 
around the control element. Preferably &e help 2D gesture has substantially the shape of 
a question mark. 

5 The control element can be visually transparent The control element can have a 

transparency of l^s than substantially 40%, preferably less than substantially 30%, more 
preferably less than 20%. The control element can have a transparency m the range of 
substantially 10% to 40%, substantially 10% to 30%, or substantially 10% to 20%. Low 
levels of visibility for the control elements enhance visibility of the background, but the 
10 aiumation and/or motion of the control elements allows a user to reliably identify the 
overlaying control element. 

Hie user interface can include a plurality of animated control elements. Each control 
element can be associated with a different region of the user interface. Each control 
15 element can be associated with a different group or set of functions, operations or 

conomands. Some of the individual operations, functions or cominands can be common 
to different groups. The 2D gestures that can be used to select and/or execute a function, 
operation or command can be the same or different for different control elements. 

20 The first control element can be of a first type and a second of the plurality of control 
elements can be of a second type different to (he first type. The type of a control element 
can be any of; its animation; its movement; or other attribute of its visual appearance, 
such as those mentioned above, e.g. a word, icon, symbol etc. 

25 The plurality of control elements can between them provide a keyboard. Each of the 
pluraUty of control elements can have a different group or set of characters or letters 
associated with them. The keyboard can have a plurality of regions. Each region can 
have a plurality of control elemrats associated with it A first control element can have a 
letter or letters associated with it and/or a second control element can have a numeral or 

30 numerals associated with it and/or a third control element can have a symbol, symbols, or 
foimattmg function, e,g. tab, space or similar, associated with it. The function, command 
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or operation associated with the control elraient can be to display selected entity on the 
background 

The keyboard can have a standard layout The keyboard can provide characters, letters or 
5 symbols in an alphabet of a language. Hie language can be any language, but is 

preferably the English language. The language can be a ideogram based language such as 
Chinese, Japanese or Korean. Preferably tiie keyboard includes all of the charters^ 
symbols or letters of a language. 

10 At least one of the control elements is associated with a plurality of characters. Each of 
the plurality of characters can have a respective 2D gesture associated therewith. The 
gesture can cause tiie character to be displayed on the background layer. 

The control element can have a 2D gesture associated with it for carrying out a formatting 
15 function on a character associated with the control element. For example, the 2D gesture 
could cause the character to be displayed underlined, in bold or having a different size or 
font The 2D gesture can be a continuous part of a 2D gesture used to select the character 
or can be a discrete gesture. 

20 The control elements can be associated with a plurality of media player functions. Each 
of the media player ftmctions can have a respective 2D gesture associated therewitih for 
causing the media player function to be executed The media player functions can 
include, play, stop, forward, reverse, pause, eject, skip and record. 

25 The control element can be animated so as to have a three dim^ional appearance 

The control element can be animated so as to be more readily noticeable by peripheral 
vision. The control element can have an axis along which it is animated. The animation 
can be configured to progress, change or vary in a certain direction. The control elements 
30 animation can comprises variable thickness bars scrolling along an axis, or in a direction. 
The control element can rotate in a plane parallel to the background The degree of 
rotation can be used to provide a dial in which the direction or animation provides a 
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pointer of &e dial. The animation of the control element can vary depending on its 
rotation, e.g. the speed of animation, the colonr of animation, the size of compon^ts of 
the animation, the nature of the animation, and similar, including combinations of the 
aforesaid 

5 

According to a further aspect of the invention, there is provided an electronic device 
including a display device, a data processing device and a memory storing instructions 
executable by the data processing device, or otherwise configuring the data processing 
device to display a user interface on the display according to any of flie first aspect of the 
1 0 invention, and including any of the aforesaid preferred features of the user interface. 

The display can be a touch sensitive display. This provides a simple pointer mechanism 
allowing a user to enter gestures using either a separate pointing device, such as a stylus, 
or a digit, or part of a digit, of the user's hand. 

15 

The device can further include a pointer device for making a 2D gesture on the user 
interface. Any suitable pointing device can be used, such as a mouse, joystick, joypad, 
cursor buttons, trackball, tablet, lightpen, laser pointer and similair. 

20 The device can be a handheld device. The device can be a handheld device having a 
touch sensitive display and the device can be configured so that a user can make 2D 
gestures on the touch sensitive display with a digit of the same hand in which the device 
is being held, hi this way one handed use of ttie device is provided 

25 The device can be a wireless telecommunications device, and in particular a cellular 

telecommunications device, such as a mobile telephone or smart phone or combined PDA 
and communicator device. 

According to a further aspect of the invention, there is provided a computer unplemented 
30 method for providing a user interface for a display of an electronic device, comprising 
displaying a background layer, displaying a control element associated with a pluraUty of 
functions over the background layer; detecting a 2D gesture made over a region of the 
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us^ inter&ce associated with the control element; and executing or selecting a function 
associated with the 2D gesture. 

The method can include steps or operations to provide any of the prefeired features of the 
5 user interface as described above. 

A plurality of animated control elements can be displayed The control elements can be 
animated and/or transparent 

1 0 Detecting a 2D gesture can comprise a gesture engine parsing the 2D gesture and 
generating a keyboard event corresponding to the 2D gesture. 

The method can fiirther comprise determining a location or region within the display or 
user interface in which the 2D gesture, or a part of the 2D gesture was made; The method 

15 can fiirther include determining whether a control element is associated with the location 
or region. The metiiod can further comprise determining whether the location or region^ 
or control element, has a particular keyboard event associated with it. The method can 
include determining which command^ fimction or operation to select of execute by 
determkdng if a region in which a gesture was made has a control element associated 

20 with it and if the keyboard event coirespondiug to the gesture corresponds to a one of the 
commands, operations or functions associated with the control element. 

The method can further comprise determining whether a gesture is intended to activate a 
control element and if not then determining or selecting a function of the background 
25 layer to execute. Determining can include determining whether a time out has expired 
before a pointer movement event occurs. 



30 



The 2D gesture can be a help 2D gesture and the function associated with the 2D gesture 
can be a help function which displays mformation relating to the control element adjac^t 
and/or around the control element 
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The information relating to the control element can include a gr^hical indication of all or 
some of aie 2D gestures associated with the control element arid/or text explaining the 
functions and/or gestures associated with the 2D control element 

5 The control element can be associated with a menu or group of functions or data items 
and the 2D gesture can cause a one of the fimctions from tiie menu or group of functions 
to be executed or to select a one of the data items. 

The plurality of control elements can between them provide a key board and the 2D 
10 gesture can cause a character, numeral, symbol or formatting control selected from the 
keyboard to be displayed on the background layer. 

The control element can be a character string and preferably the character string is a word. 
The word can be a polysyllabic word and each syllable of the word can be separately 
15 animated. 

According to a ftirther aspect of the invention, there is provided computer program code 
executable by a data processing device to provide the user interface aspect of the 
invention or the computing device aspect of the invention or the method aspect of the 
20 invention. According to a further aspect of the invention a computer program product 
comprising a computer readable medium bearing computer program code according to the 
preceding aspect of the invention is provided. 

An embodiment of the invention will now be described, by way of example only, and 
25 with reference to the accompanying drawings, in which: 

Figures 1 A to ID show graphical representations illustrating the constraints 
imposed by combining a keyboard and text area on a siugie display device; 
Figure 2 shows a diagrammatic representation of a control element part of the 
30 user interface of the present invention and an associated 2D gesture; 

Figure 3 shows a diagrammatic representation of an overloaded user interface 
according to the present invention; 
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Figure 4 shows a schematic hlock diagram of a device including a user inter&ce 
according to the invention; 

Figure 5 shows a high level process flow chart illustrating a computer program 
providing the user interface according to the invention; 
5 Figures 6A to 6C show a mohile phone mcluding a user inter&ce according to 

the present invention illustrating use of the user inter&ce by a user; 
Figures 7A to 7B show different screens of the user interface of the phone shown 
in Figures 5A-5C illustrating further functionalities of the user interface of the 
invention; 

10 Figure 8 shows a process flow chart illustrating parts of the flow chart shown in 

Figure 7 in greater detail; 

Figure 9 shows a diagrammatic representation of a control element layer and 
background layer of the interface illustrating selection of a control element of the 
background layer; 

15 Figure 10 shows the mobile phone shown in Figures 5 A to 5C displa3dng a 

keyboard part of the user int^face according to the present invention; 
Figure 1 1 shows the keyboard part of the interface shown in Figure 10 in greater 
detail ilhistrating animation of the keyboard control elements; 
Figures 12 shows a diagrammatic representation of the overloading of a set 

20 media player controls onto an overloaded control element part of the user 

inter&ce of the invention and the associated 2D gestures; 
Figure 13 shows a graphical representation of a help function invoked by a 2D 
help gesture being applied to the overloaded control element of Figure 12; and 
Figure 14 shows a process flow chart illustrating execution of the help operation 

25 which has been invoked as illustrated in Figure 13; 

Figure 15 shows an overloaded control element part of the user interface of tiie 
invention adapted for peripheral visibility. 



Similar items in different Figures share common reference numerals imless indicated 
30 otherwise. 
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Before describing some preferred embodiments of the invention, a discussion of the 
requirements of a user interfece, taken into account by the invration, will be provided 

Two examples can be used to illustrate the trade off between redundancy, ergonomics of 
5 use and visible display. A full screen keyboard allows direct manual interaction due to 
larger keys and a c£5)acity for more keys but at the expense of display real estate. 
Secondly, the standard split screen keyboard afready limited in size, sacrifices redundant 
controls to permit larger keys and to make more visible ^splay available. However, its 
small size results in the need to use an additional device, such as a stylus, which results in 
10 an approach that is difficult to use dextrously with die digits, Le. fingers or tiiumbs. 

The present invention appreciates that a problem with many text input solutions is the 
lack of appreciation of the true difficulty with handheld device text input What is 
important is not the mechanism for inputting text in itself, but rather the consideration of 
15 the constraints on inputting, such as constraints on the available size of a text mput panel 
and free display area, 

' With reference to Figures 1 A to ID there are respectively shown schematic illustrations 
of four keyboard and display area configurations 102, 104, 106 and 108 illustrating the 

20 constraints on a keyboard and display based user interface. The first configuration 102 
has a small display area 1 10 and a large keyboard area 1 12, witih small keys. The second 
configuration 104 has a small display area 1 14 and a large keyboard area 1 16, with large 
keys. The third configuration 106 has a large display area 118 and a small keyboard area 
120, with large keys. The fourth configuration 108 has a large display area 122 and a 

25 small keyboard area 124, witii small keys. 

The layout of a command and text input mechanism is subject to some physical 
constraints which affect usability. In order to firee up as much screen display as possible, 
input dialogues can be reduced in size (FIGS. IC & ID), which reduces tiie size of 
30 individual keys, making them more difficult to select. Increasing the number or 

redundancy of controls limits the space available. The size of keys is also subject to the 
number of keys on the keyboard. A large number of keys means less space per key 
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(FIGS. 1 A & ID), or a smaller input text panel (FIGS. 1 A & IB). Alternatively, to 
minimise the display area used by Ihe keyboard, and maintain a reasonable sized key, a 
designer can use menus or modes. Seldom used commands inevitably feature in 
submenus, which leads to a slow and awkward interaction approach. 

5 

These constraints are subject to the constraints defined in Fitts' law: a large dialogue is 
subject to a time overhead firom increased hand travel, while smaller keys take up less 
space and merit a reduced hand travel, yet may incur a time overhead due to a fine motor 
control requirement in selecting a key. Overly small keys result in eith^ unacceptable 

1 0 increases in error rates or unreasonably slow input rates for text input, due to 

awkwardness of selecting a key accurately. This suggests a larger keyboard should be 
favoured- 
Ancillary pointers, such as a stylus, clip on keyboards and data gloves, can impede device 

1 5 usability. To interact with the device the user must either don the interaction accessory 
or, say, pick up a stylus, which in the case of many portable devices, ties up both hands. 
Therefore a more preferred interfece would allow one handed use of the device and 
interface. However, die invention can also be used with a stylus, mouse or other pointer 
device. 

20 

Many prior small device text input approaches are not easily leamed. The user expends 
time to learn numerous gestures and the different contexts they can be used in. . 

Drawing fi:om the above evaluation of text input solutions a definition of the design 
25 requirements can be constracted, and which is fulfilled by the approach of the present 
invention, rather than merely further optimising on approaches that &il to address 
relevant issues such as screen real estate or convenience of use. For example the over 
engineered optimisations of conventional soft keyboards. 

30 Consideration of the contributing factors in the design of interaction models for handheld 
and mobile devices leads to the following design considerations. Larger keys for manual 
interaction should be favoured over interaction aids. For example styluses, obstruct the 
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freedom of a hand, posing a hindrance to handheld interaction. A good balance should be 
sought between redundancy in the number of visible input device features and availability 
of display area. An effective trade-off between display area, size of elements in the input 
panel, and usability should be provided. The approach should be easy to leam to use and 
5 understand or there should be a justifiable benefit for any learning overhead. 

The user interface of tiie present invention is based on a system of interaction for entering 
commands, instructions, text or any other entry typically entered by a keyboard, pointing 
device (such as a mouse, track ball, stylus, tablet) or other input device, whereby a us^ 
10 can selectively interact with multiplexed or visually overloaded layers of transparent 
controls with the use of 2D gesture. 

A control, or control element, can be considered flmctionally transparent in the sense that 
depending on the gesture applied to the control element, the gesture may propagate 

15 through the control element, and operate a further element on a background layer on 
which the control element is overlaid, or not. For example is a gesture is one that is 
associated with the control element, then a fimction associated with the control element 
may be executed. If the gesture is not one associated with the control element, e,g. a 
mouse "point and click' gesture, then an operation associated with tilie underlying element 

20 of the backgroudn may be executed 

Visual transparency has been used previously in user interfaces, e.g. to display a partially 
visually transparent drop down menu over an application. This transparency has been 
used to optimize screen area, which can often be consumed by menu or status dialogues, 
25 The aim is to provide more visual chies m the hope the user will be less likely to lose 
focus of theu" current activity. However, this approach of using a layer of transparency to 
display a menu is done at the cost of obscuring whatever is in the background. This is not 
actually visual overloading, but rather a con5)romise between two images competing for 
limited display area. 

30 

In terms of visual appearance, ttie control element itself may be rendered and displayed 
either in wholly visually opaque form, or a partially visually opaque form, in which parts 
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of the control element are opaque, but parts are transparent so that a user can see the 
underlying back ground layer Additionally, the control element itself may be rendered 
and displayed in an at least partially visually transparent form, in which elemrats of the 
background layer can be seen through the control element. 

5 

2D gesture will generally be used herein to refa: to a stroke, trace or path, made by a 
pointing device, including a user's digits, which has both a magnitude and a sense of 
direction on the display or user interface. For example, a simple "point and click' or stylus 
tap will not constitute a 2D gesture as those events have neither a magnitude nor a 

10 direction. A 2D gesture includes botii substantially straight lines, curved lines and a 
continuous line havmg straight and curved portions. Generally a 2D gesture will be a 
continuous trace, stroke or path. Further, for pointer devices allowing a 3D gesture to be 
carried out by a user, that 3D gesture can also result in an at least 2D gesture being made 
over the display device or user interface and the projection of the 3D gesture onto the 

15 display device or user interface can also be considered a 2D gesture, provided it amounts 
to more than a simple 'point and click' or tap' gesture. 

Visual overloading is different fiom the use of static layered transparencies. An 
embodiment of the present inv^tion renders an animated image or a transparent static 
20 image panel wiggling over a static background, which will visually multiplex or visually 
overload the overlapping images. The result is that a layer of controls appears to float 
over the interface without interfering with the legibility of the background. Overloading 
can be achieved to some degree using both ^preaches on an animated background. 

25 The use of 2D of gestural input provides a mechanism by which to resolye the issue of 
layer interaction. Gesture activation has been used previously, for example with marking 
menus, but this approach only uses simple gradient stokes or marks and not with 
transparent control elements. Further, the present invention also makes use of more 
sophisticated gestures. The underlying principle of marking menus is to facilitate novice 

30 users with menus while offering experts a short cut of remembering and drawing the 
appropriate mark without waiting for the menu to appear. In contrast, the present 
invention uses 2D gestures for selective layer interaction. That is any one of a plurality of 
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functions or operations ("layers") associated with a particular control elem^t can be 
selected by applying a particular 2D gesture to the control element which selects and 
activates the corresponding operation or layer, 

5 This approach of incorporating 2D pointer gestures to activate commands associated with 
a control, provides the necessary additional context required beyond that of the restricted 
pomt and click approach. This enables the user to benefit from the added properties 
associated with an overloaded control by enabling the selective activation of a specific 
function related to a control contained in the layers. 

10 

For example. Figure 2 shows a diagrammatic conceptual representation of an overloaded 
control element 1 30 which can be used in the user interface of the present invention. The 
control element itself has three "layers" 131, 132, 133 each of which is associated with a 
particular function graphically represented in Figure 2 by a diamond, square and triangle 

15 respectively. The background or underlying layer 134 of the user interface, over which 
HiQ control element is overlaid, can also have a function associated with it as illustrated 
by the oval shape in Figure 2. The shapes shown in Figure 2 are merely by way of 
distinguishing the different functions associated with the different layers and are not 
themselves visually displayed. Rather, a single control element is displayed ovrar the back 

20 ground 134 layer and any one of the three functions associated with the control element 
can be selected by making the appropriate 2D gesture associated with the function over 
the control element. 

For example, as ilhistrated in Figure 2, by making a "T" shaped 2D gesture 135 over the 
25 part of the display associated with the control element 130, die triangle function Le. the 
function associated with the third layer 133 of the control element can be selected and 
executed. For example, the control element could be an animated folder overlaid over the 
user iuterj&ce for an application, such as a word processor or spread sheet application. 
Hence the folder will provide file handling functions- For example, the first layer could 
30 be associated witti an open file function, the second layer with a close file function, the 
third layer with a delete file function and the application interface or background layer 
could be associated with some other fimction of the application, e.g. a printer operation. 
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Hence by executing an upper or lower case 0, D or C shaped gesture over (he control 
element the file open, file delete or file close operations can be called and executed 

In another example of an animated control element^ more than one item can be 
5 represented in the same area as part of a media clip. For example, a triangle could change 
into a circle, and then into a rectangle and finally into a trapezium. This provides a 
thematic representation. The event of tiie change is remembered by a user, allowing all 
items to be recalled as one event contained in one area. 

10 Hence, the present invention pemiits the intensive population of a display through the 
layering of control elements. This can be achieved without compromise in size of the 
inputted text panel or to the size of control elements. This approach effectively gets 
round the constraints described earlier by permitting background and subsequent layers to 
occupy the same screen real estate. 

15 

For example. Figure 3 shows a diagrammatic representation of a user interface 140 
combining an overloaded keyboard layer 142 over a back groimd text display layer 144. 
Each of the keys of the keyboard can be in the form of a control element so that one of 
multiple operations can be carried out by making the appropriate 2D gesture over the 

20 region of the display associated with each key. For example a first 2D gesture on a key 
could cause a first character to be displayed on the underlying text layer, a second 2D 
gesture on the same key could cause a symbol to be displayed on tiie underlying text 
layer, and a third 2D gesture on the same key could cause a numeral to be displayed on 
the underlying text layer. Another control element 146 having two layers 147, 148 or 

25 Amotions associated with it can also be provided as an animated icon or symbol over the 
keyboard layer 142. For example control elemmt 146 could have an 'email' fimction 
associated with the first layer 147 and a 'send to printer* function associated with the 
second layer 148. Hence, making the appropriate 2D gesture, e.g. an upper or lower case 
'e' or 'p', over the display region associated with the control element 146 would select and 

30 execute a fimction to either e-mail or print the text on the underlying text layer 144. 
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Another benefit is the availability of real estate pennittiiig larger controls, which are 
easier to locate, improving input rates and fecilitate manual interaction. 



Constraints of this approach are that too many elements can gradually cause the 
5 background to lose coherence, /. e. obscures the background, or the interface can become 
visually noisy if too miany layers are added. However appropriately chosen layers permit 
a reasonable number of controls to be provided before this constraint takes effect 



Hence, the present invention eliminates the constraints between the size of the display 
10 and the input dialogue. In addition the redundancy of a control can be increased in a new 
way, by overloading the functionality of a control with a selection of gestures, diereby 
avoiding the use of obtrusive context menus. 

An example embodiment of the invention in the form of a user interface for a cellular 
1 5 telecommunications device, such as a mobile telephone or mobile smart phone will now 
be described. 

Figure 4 shows a schematic block diagram of the computing parts of an electronic device 
200. Those parts of the mobile phone device relating to its communications functions are 

20 conventional and are not shown so as not to obscure the nature of the present invention. 
Furth^ the present invention is not limited to communications devices and can be used in 
any electronic device having a screen and which may benefit firom the use of a user 
interface. Further, electronic devices are not considered to be limited only to devices 
primarily for computing, but is considered to include any and all devices having, or 

25 including, sufficient computing power to allow the present invention to be implemented 
and which may benefit from the user interface of the present invention, e.g, vehicle 
control systems, electronic entertainment devices, domestic electronic devices, etc. 

Electronic device 200 includes a processor 202 having a local cache memory 204. 
30 Processor 202 is in conununication with a bridge 106 which is in torn in communication 
with a peripheral bus 208. Bridge 206 is also in communication with local memory 210 
which stores data and instructions to be executed by the processor 202. A mass storage 
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device 212 is also provided in communication with the peripheral bus and a display 
device 214 also communicates with Ihe peripheral bus 208. Pointing devices 216 are also 
provided in commimication with the peripheral bus« 

5 The pointing device can be in the form of a touch sensitive device 218, which in practice 
will be overlayed over display 214. Other pointing devices, generically indicated by 
mouse 220 can also be provided, such as a joy stick, joy pad, track ball and any other 
pointing device by which a user can identify positions and trace paths on the display 
device 214. For example in one embodiment, the display device 214 can be a data board 

10 and the pointing device can be a laser pointer with which a user can identify positions and 
trace paiiis on the data board. In o&er embodiments, the display device can be a three 
dimensional display device and the pointing device can be provided by sensing the 
positions of a user's hands or other body part so as to "point" to positions on the display 
device. In other embodiments, the position of a user's eyes on a display can be 

15 determined and used to provide the pointing device. However, in the following 

exemplary discussion, use of a mouse and a touch sensitive display will in particular be 
described. However, Ihe invention is not intended to be limited to this particular 
embodiment 

20 Bridge 206 provides communication between the other hardware components of the 
device and the memory 210. Memory 210 includes a first area 222 which stores 
inpul/output stream in&rmation, such as the status of keyboard commands and the co- 
ordinates for pointer devices. A further region 224 of memory stores the operating 
system for the device and includes therein a gesture engine 226 which in use passes 

25 gestures entered into the device 200 by the pointing device 21 6 as will be described in 
greater detail below. A further area of memory 228 stores an application having a user 
int^r&ce according to the invention. The application 228 also includes code 230 for 
providing the graphical user inter&ce of the invention. The user interface 230 includes a 
system event message handler 232 and code 234 for providing the overloaded control 

30 elements of the user interface 230. Application 228 also includes a control object 236 
which provides the general logic to control the overall operation of the apphcation 228. 
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The graphical user interface 230 can be a WMP (Windows/icons/menns^ointers) based 
interface over which the control elements are overloaded. The system event message 
handler 232 listens for specific keyboard events, provided by the gesture engine 226. The 
system event message handler 232 also listens out for pointer events falling within a 
5 region of the display associated with a control element The control element overloading 
module 234 provides a transparent layer, including tibie conhrol elements, over the 
conventional part of the user interface. The transparent layer is implemented to allow ttie 
animated transparent control element to be rendered over the controls of Ihe underlying or 
background layer. This can be achieved by either creating a window application using C# 

10 with an animated icon and specifying a level of opacity, or, as with some languages, such 
as J# and Java, a glass pane can be layered over a regular inter&ce* Another way of 
implementing the animated control elements is to write the individual images comprising 
the animation (e.g. 25 fii^es) into different memory addresses in a memory buffer and 
then alpha-blending each of the frames from the memory over the background user 

15 interface layer. 

In one embodiment, the application can be written m the Java programming language and 
executed using a Java vhtual machine implementation, such as CREAM. A suitable 
gesture engine would be the Libstroke open source gesture engine. Alternatively, the 
20 overloaded control element module can be written in C#, for example, and using a low 
opacity setting in order to generate the animated control elements from the individual 
frames of the animation stored in memory, layered on top of bespoke standard controls, 
e,g. buttons. 

25 With reference to Figure 5, there is shown a high level process flowchart illustrating the 
computer implemented method 250 of operation of the device 200. Processing begins at 
step 252 and at step 254, the device is initialised, which can include initialising the 
gesture engine and otherwise preparing flie device for functioning. Then at step 256, the 
control elements are initialised. This can iaclude, for example, writing the frames for the 

30 animated control elements into memory areas, ready for display. Then at step 258, tiie 
underlaying background WMP based user interface layer is displayed and the control 
elements are displayed over the background layer and their animations begun. 
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Willi reference to Figures 6A, 6B and 6C, there is shown a device 200 including an 
example of the user interfece 270 of the present invention. The user interface 270 
includes the background layer interface 272 and a first transparent animated control 
element 274, being an icon in the form of an envelope, and a second animated transparent 
5 control element 276 in the form of the word "register". Each of the control elements, 274, 
276 has a separate area of the user inter&ce 270 associated with them. 



Figures 6A, 6B and 6C show different screen shots of the same user interface so as to try 
and illustrate the animation of the control elements. The control elements are animated in 

10 Ihe sense that their form, that is tiieir appearance or shape, changes rather than merely 
moving over the display. However, the mvelope control element 274 also moves over 
the display and sunilarly parts of the register control element 276 also move, and also 
vary in size. Each of the syllables of the register word changes separately that is the re 
syllable shrinks and grows and moves over the screen, the gis syllable shrinks and grows 

1 5 and moves over the screen and the ter syllable shrinks and grows and moves over the 
screen individually. However, these three elements together provide the overaU control 
element 276. 



As can be seen, the control elements 274, 276 are visually transparent as the background 
20 interface can be seen through the control elements. However, portions of the control 
elements, e. g. lines or individual characters, are themselves opaque, alfliough in other 
embodiments those parts can also be transparent Such animations are sometimes 
referred to as animated transparent Qi& in the art. A particular colour is made transparent 
and therefore using it as the background colour leaves an image clipped to the outline of 
25 the image. Another way of providing transparency is to use alpha-blending as is 
understood in the art 



Returning to Figure 5, at step 260, the ^plication detects whe&er a gesture has been 
applied to the user interface by a reporter device. In the illustrated embodiment, the 
30 device 200 has a touch sensitive screen and the interaction of a user's digit and the touch 
sensitive screen provides the points device. As illustrated in Figure 6A, a user can tap 
the screen on the answer phone menu option of tiie underlying display and at step 262, the 
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answer phone preparation can be executed. Process flow then retoms, as illustrated by 
line 264, to step 260 at wtuch a fur&er gesture can be detected. 

In order to invoke a one of the fimctions associated with a one of the control elements, the 
5 user m^es a two dimensional gesture over the part of the user interface associated with 
Ihe control element. Examples of the kinds of gestures and functions that can be executed 
will be provided by the discussion below. At some stage, the user can enter a gesture^ 
either a conventional "point and click" gesture or 2D gesture in order to temainate flie 
application and processing ends at step 226. 

10 

Commands can be executed in the user interne 270 with either standard "point and 
click" over a list item or iho user can circumvent the intrusive hierarchical menu 
interaction approach by drawing a symbol (2D gesture) that starts over the relevant list 
item, which takes the user directly to the required dialogue or executes the desired 
15 conmiand. Note that a stroke or 2D gesture is not restricted in size. 

In addition, the overloaded layer of control elements is placed over the back ground menu 
items and control clients. A control or command fiom one of the layers within a region 
of the overloaded control can be selected with an appropriate gesture, thus 
20 disambiguating between competing controls and menu items. This permits a larger 
population of control elements with an adequate degree of redundancy, yet without 
compromise to tbe size of control elements or menu. 

Simple animated black and white transparent gifs can be used to implement the control 
25 elements. Adequate performance is possible without alpha blending, although tibiat can 
improve the user interface perforaiance. Simple well chosen animations can be as 
important as the transparency. 

Use of the interfece 270 shown in Figures 6A to 7E various interaction scenarios will now 
30 be described to help explain the use and benefits of the interface of the invention. 
Interacting with the interface 270 is straightforward. As illustrated in Figure 6A, the 
interfece 270 in Figure 6A has a Ust of frequently called numbers, two overloaded icons, 
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one for messaging functions 274 and one for accessing *call register* functions 276, witii 
two gesture optimized control elements 278, 280 in the form of MENU md a NAME 
buttons respectively at the bottom of the display items. 

5 To access a list element the user can either tap over it or gesture over it For example, 
from the list of frequently used numbers (Figures 6A-6C) in the background mterface, or 
a generated list of names, to access the details of a telephone number the user can click on 
the list element to access a submenu and select a 'get details' command from a list of 
options. Alternatively, as depicted in Figure 6B, the user can simply draw a 'd' gesture 
1 0 starting over the list element, to go straight to the desired list details** dialogue, in this 
case from the item marked ^sport centre\ 

In order to populate the display with more controls without compromise to manual 
interaction and tiie size of control elements in the backgroimd interface, the interfece 270 
15 has two overloaded icons or control elements 274, 276. Again, executing the appropriate 
gesture over a list item will execute a command. However, if the gesture starts over any 
list element that lies in a region associated with an overloaded control element icon and 
the gesture relates to that overloaded control element icon, then the command 
corresponding to that gesture is executed 

20 

For example, drawing an 'M* stroke 282 over the 'register' overloaded icon 276, 
demonstrated in Figure 6A, accesses a ^Missed calls' dialogue, whereas executing an *r* 
gesture accesses a 'Received calls' dialogue. 

25 This form of interaction model is not restricted to gestural interaction alone; more 

conventional *point and click' or *tap' gestures can be used when required, such as when 
dialling a number (see Figure 7B), or, in Figure 6A, where a double tap on a list element, 
rather than drawing a 'd', will call flie selected number. 

30 Figure 7A ilhistrates the use of a 2D gesture driven button 278. Simply drawing an 
upward line 2D gesture 284 invokes the dialogue to enable dialling, avoiding any sub 
menu interaction (see Figure 7B). Altematively, simply tapping on the *Menu' button 278 
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will enable the user to access a hierarchical memi, as in conventional interfaces, 
containing an option to 'Dial a number'. This approach demonstrates the practical 
integration of the two modes of interaction. 

5 Figure 7C illustrates the use of the gesture activated **Name" button 280 to search for a 
given phone number. By drawing a 'T' shaped gesture 286 the list is set to and displays 
all elements that begin with the letter T' (Fig. 7D) and by drawing a shaped gesture 
288 (middle) the list is further optimized to all elements that begin with the letter *T' and 
contain flie letter *P\ This approach drastically cute down on executions for selecting a 
10 letter, whilst possessing a greater coguitive salience. 

Drawing a symbol or tapping on the left of tiie list 290 executes a command; such as a 
double-click to call a number. Moreover, a symbol drawn on the right side of the list 290 
will forfher refine the search to any remaining items tihat contain tiie desired letter. To 
15 access an element the users can again either tap on an item or gesture appropriately over 
the relevant list iteuL 

With reference to Figure 8, there is shown a flowchart illustrating the data processing 
operations carried out in order to handle the gesture based input to the user interface 270, 

.20 and correspondingly generally to steps 260 and 262 of Figure 5. The process 300 begins 
at 302 and at step 304, the gesture engine 226 intercepte gestures inputted by the pointing 
device, be it either a mouse entered gesture, touch screen entered gesture or from any 
other pointer device* The gesture engine passes the gesture and at step 306 determines a 
keyboard event which is associated with the gesture. The gesture engine outputs the 

25 keyboard event and at step 308, the user mterface handler 232 intercepts the keyboard 
event and any pointer event and the current pointer co-ordinates. A pointer event, in this 
context, means a control command indicating that a pointer has been activated, e.g. a 
mouse down event or a "tap" event on a touch screen. 

30 Then, step 310 discriminates beftveen pointer evente which should be passed through to 
the underlying interface and any pointer events that are intended to activate a control 
element In particular, at step 3 10, it is determined, using the pointea: co-ordinates. 
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whether the pointer event has occurred within a region associated with a control element 
and if so, whether a gesture has begun within a time out period Hence, if a pointer event 
is detected in a region associated with the control element but there is no motion of the 
pointer device to begm a 2D gesture within a fixed time period, then it is assumed that the 
5 command is intended for the underlying layer. 

This first scenario is illustrated in Figure 9 which shows a diagrammatic representation of 
distinguishing between pointer events intended to invoke an overloaded control element 
320 or a control element of the underlying background layer 322. A static cursor 324 
10 illustrates a mouse down or "tap" pointer event which is not followed by movement of the 
pointer and so a control element 322 in tbe underlying interface 326 is invoked 

Returning to Figure 8, in this scenario, the user interface event handler 232 makes a 
system call passing the event to an event handler for the underlying layer 326. Then at 
1 5 step 320, the event handler for the underlying layer handles the event appropriately, e.g. 
by displaying a menu or other dialogue for executing an appropriate function. The 
process then completes at step 322. 

Retuming to step 310, if pointer movement is detected within the time out period, as 
20 illustrated by cursor 328 tracing a gesture 330 over a region of the user int^ace 
associated with the control element 320^ then this pointer event is detemiined to be 
intended to invoke a overloaded control element 

Process flow proceeds to step 312 at which it is determined in which of the regions of the 
25 display associated with overloaded control elements, the pointer event has occurred, hi 
this way, it can be determined v/idch of a plurality of control elements, the 2D gesture is 
intended to have invoked Then at step 314, it is determined which of the pliu:ality of 
commands associated with the control element to select In particular^ it is determined 
whether the keyboard event corresponding to the gesture is associated with a one of the 
30 plurality of commands for the control element in that region and if so, then at step 316, 
the selected one of the plurahty of commands, operations or functions is executed. 
Process flow then terminates at step 324. 
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If at step 314, it is determined that &ere is no command associated with tiie keyboard 
event corresponding to the gesture applied to the control element (e.g. there is no 
command associated with an 'X' shaped gesture) then process flow branches and the 
process 300 t^minates at step 326. 

5 

Hence the overloaded control elem^ts can be integrated seamlessly with WIMPS 
offering extended functionality by intercepting gestures but allowing standard point and 
click interaction to pass through the layers where they are handled in a conventional way. 
Such a user interface could interfere with drawing packages and text selection. However, 
10 the solution to this is to avoid conflicts using a small time delay to switoh modes as 
described above or alternatively to use die right mouse key to activate gesture input 

It has been found that overloaded transparent control elements work with very low levels 
of transparencies, lower than the 30% opacity for static images typically suggested. 

15 

Other restrictions which exist and that can be avoided with good design are, the choice of . 
colours conflicting widi the background, and in the poor choice of animations which may 
result in difficulties selecting moving elements or distinguishing between layers. 
However, this is no more an overhead tiian in designing graphics for a standard interface 
20 or web site. Another restriction is animated controls can be obscured on a moving 
background, such as a media cUp. 

Referring back to Figure 6A drawing a 'C* over the animated envelope opens a text input, 
or compose, dialogue 350 (Figure 10) including an overloaded keyboard 360 shoAvn in 
25 greater detail in Figures 1 1 A, 1 IB and 1 IC, whereas an or *0' would invoke an 

*hibox' and *Outbox', respectively. The text input or "Compose" dialogue makes use of 
an overloaded layer of text, in the same style as that of the 'Register' overloaded control 
element icon 276 from the initial screen (Figures 6A-6C). 

30 The keyboard 360 is implemented as a visually overloaded ISO keyboard layout (standard 
on mobile phones) and a number pad layered over the text 2D gestures are incorporated 
using simple gradient strokes to select a letter and simple meaningful gestures to access 
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Other functions^ such as numbers and upper case letters. An array of nine transparent 
green dots 361 provides a visual clue as to the nine areas on the display having control 
elements associated therewith. A group of transparent characters 363, e.g. three or four, 
in a first colour, e.g. blue, are animated and gradually grow and shrink in size as they 
5 move over a region of the display near the associated green dot Animated numerals 364 
are also associated with green dots and a transparent numeral in a second colour, e.g, 
blue, is simDarly animated and grows and shrinks in size and moves around a region of 
the display near the associated green dot. Similarly animated punctuation marks 365, or 
other symbols or characters^ are also associated with green dots and transparent symbols 

10 or characters are similarly animated and grow and shrink in size and moves around a 
region of the display near the associated green dot The background layer then provides a 
display for the text 362 entered by the keyboard as described conceptually above with 
reference to Figure 3. Hence, Figures 1 1 A-1 IC show three frames of the animated 
keyboard 360 which is made up of a plurality of overloaded control elements each having 

15 an associated region. 

To operate the keyboard (see Fig. 10); the user makes very simple gradient gestures, e,g. 
370. To select a letter, a gradient stroke that starts over the selected button is performed 
The centre point of a button is indicated with the green dot The angle of a gesture 
20 supplies the context indicating which element is being selected. "L" would be selected 
with a right terminating gesture 370, as shown in Figure 10, while '*K" would be selected 
with a vertical up or downward stroke. To improve usability the "space" character is 
selected with a "right-dash'' gesture, that can be executed anywhere on the display. 
Similarly a delete command is selected with a global "left-dash". 

25 

To access lesser used fimctions, other than basic text input, the approach uses more 
elaborate 2D gestures such as selecting the number "5" with a meaningful and easily 
associated 'V gesture made in the region of the keyboard associated with the 5 numeral 

30 Other options include clearing text from the underlying display of the screen with a "C" 
gesture and a capital can be entered by drawing a "U" for upper case either immediately 
after, or as a continuous part of the 2D gesture for, the desired letter. The need to learn 
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th^e associations does pose some learning overhead, however they can easily be learned 
especially using tilie help mechanism to be described below. Initially, this use of symbols 
is no less awkward than selecting a mode or menu option, however as the operation 
becomes famihar, it ceases to be as obtrusive as the other approaches. Point and click 
5 interaction is left alone to demonstrate that the approach could mcorporate the T9 
approach and could still use standard text interaction, such as with text editing m 
conventional graphical interfaces. 

A further option is to use the length of a gestm-e to indicate the length of a word as part of 
10 a predictive text input mechanism. For example, the initial letter of a word is entered via 
the keyboard with the appropriate 2D gesture and then the user makes a gesture the length 
of whidi represents the length of tiie word. The predictive text entry mechanism then 
looks lip words in its dictionary beginning with the initial letto: and having a word length 
corresponding to the length of the gesture and displays those words as the predictions 
15 from which a user can select. The 2D gesture identifying the word length can have the 
general shape of a spike, or pule, similar to the trace generated by a heartbeat monitor. 

The above approach to text ii^ut enables the user to enter text easily without conaplex 
combinations of keystrokes via an adequately sized soft keyboard. The benefits of this 

20 proposed design of a mobile phone interface include the following: practical manual 
touch screen interaction; the optimisation of limited screen real-estate; reduction in the 
cognitive overhead of a visual search schema, e.g., scanning for the correct button; a 
greater cognitive purchase afforded by the gesture interaction; reduction in the use of 
memory intensive sub menus, dialogues and excessively hierarchical command 

25 structures; the selection of a phone number witiiin 1 to 3 executions, raAer than Ihe usual 
3 - 8+; tiie selection of frequently used options all within one execution of a gesture, 
rather than multiple button presses; the incorporation of standard point and click 
interaction witii the optimized gesture interaction exploite redundancy of interaction 
styles. 

30 

Figure 12 shows a forther overloaded control element 380 suitable for use in the interface 
of the invention. The control element can be used to operate a media play^ device and 
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the single overloaded control element witii a ffoup of 2D gestures 382 can replace the 
five icons or control elements 384 conventionally required The control element can be 
animated so that it changes its form and can move over a region of a display on wMcli a 
user is focussed, eg the interface of an application such as a word processor. Hence (he 
5 user can easily control a media player by executing an appropriate one of the 2D gestures 
382 so as to invoke the rewind, forward, P^ay, pause or stop functions without having to 
move their visual field from their current focus. 

Figure 13 shows a graphical illustration of a help function which can be invoked by 
1 0 executing a 7* shaped 2D gesture 390 over a control element 380. A problem of gesture 
intemction is the steep learning curve, because of the need to be ^miliar wilfa a multitude 
of gestures and their contexts. The present interface supports leamability by introducing 
a mechanism wherein an easily remembered gesture will prompt the interface to 
display the gestures 382 associated with a control 380. In this way the user can become 
1 5 familiar with the system gradually, summoning help in context and when needed. This 
help functions also provides a mechanism to support goal navigation and exploration. 

To improve the usability, after the help fimction has been invoked, then a function of the 
control element can be activated in a number of ways. The user can make the correct 2D 
20 gesture over the control element or can make a pint and click or tap gesture on text labels 
or buttons 392 which are also displayed adjacent the control element. In addition a 
straight-line gesture from the control element icon 380 to tiie label 392, can be used to 
execute the operation. Hie shaped gesture may or may not require the and 
preferably does not, as illustrated in Figure 13. 

25 

Figure 14, shows a flow chart illustrating the data processing operations carried out when 
the help function relating to a control element is invoked. The overall handling of the 
pointer device event is the same as that described previously with reference to Figures 5 
and 8. The process 400 begms at step 402 and at step 404 a shaped gesture is detected 
30 over a control element Then at step 405, all of the 2D gestures 382 associated with the 
control element 380 and controls 392 labelled with the functions' are displayed adjacent 
and around the control element. At step 406 it is detennme in what manner tiie user has 
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selected to execute a one of the functions. The user can apply a 2D gesture to tibte control 
element, or draw a maxk from the control element to a labelled control or click on a one 
of the labelled control. If not of these command entry mechanisms are detected then 
process flow returns 408 to step 405 to await a correct command entry. Then at step 410 
5 the command selected by a one ofthe correct entry mechanisms is executed The help . 
process 400 then terminates at step 412. 

Figure 15 shows a further example of a control element 420 which can be used in the user 
interface of the present invention. This control element 420 is adapted to be easily 

10 distinguishable by a users peripheral vision and so can be placed in a user interface in a 
peripheral region rather than in tiie usei^ main field of view. By carefolly choosing the 
animation of the control elemmt die frinctionalityr can be improved by reducing its 
intrusiveness and elegantly increasing the prominence of the control element Animated 
control elements efifectively broaden the visual field Control elements that can be 

15 interpreted with peripheral vision, facilitate unobtrusive redundancy and the adaptivity of . 
smart interface controls. This approach thus improves the functionality of an adaptive 
mechanism by easing its intrusiveness and elegantly increasing the prominence of control 
elements. 

20 The peripherally interpretable control element 420 shown in Figure 15 is a device 
consisting of an animated transparent graphical layer that features alternating bands of 
light and daik colonr progressing over its sur&ce. The thickness of the bands vary as tiiey 
progress along an animation axis 422 of the control element. The orientation of the 
device is indicated by the direction of the progressive bands of light and dark along the 

25 ammation axis of the control element The control element can also rotate as illustrated by 
arrows 421. The animated bands provide a sense of orientation or direction of the control 
element The control element can be used to provide a "dial*' by using the animation axis 
as a "pomter*' and wherein the control element rotates, to the left or ri^t, so as to indicate 
a change in a condition. 

30 

This control element is suited to inteipretation via peripheral vision. Users have little 
difficulty reading the control element throng the comer of their eye. The user can quite 
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easily view the background and the superimposed control element 420 which eliminates 
the cognitive interruption associated with the redirecting of gaze. Thus, the field of vision 
of the user is effectively broadened. This could be particularly useful for an in car 
navigation system or speedometer, a download progress indicator or even status indicator 
for a critical system or computer game. 

A turther control element can be provided which has a cognitively ergonomic design 
heuristic, which avoids intermptions of attention caused by intmsive dialogues that often 
obscure the underlying display. For exanq)le, conventional submenus cause a high short- 
term memory load through the obscuring of the underlying work context and tiie visual 
search overhead when tiie user is required to select from a large list of options. A control 
element can be provided that iieduces both memory load and visual seaming of items by 
providing a menu system wherein drawing a letter over a menu control element, such as 
menu title or menu button, collects all the commands from that menu beginning with the 
appropriate letter. For example drawing an "o" gesture over a file menu control element 
would collect togeth^ and display all commands or functions beghmhig wilh '*o" in that 
menu. Hence, tiie system groups these commands together in a smaller, easier to handle, 
menu which is displayed to the user. Ih some cases tiiere may only be one item in the list, 
thereby dramatically reducing the necessaiy visual search. Hence, this control mechanism 
effectively has a built in search functionality. 

A further approach to improving the visual distmguishability of the control elements is to 
animate the control elements so that they appear to be three dimensional entities. This 
can be achieve in a number of ways. For example, a control element can be animated so 
that it appears to be a rotating three dimensional object, e.g. a box. Alternatively, shading 
can be used to give the control element a more three dimensional appearance. This helps 
the human visual system to pick the control element out &om the 'flat' background and 
also allows tiie control elements to be made more transparent than a control element that 
has not been adapted to appear three dimensional. 

A further control element that could be used in the user interface of the present invention, 
is a control element for providmg a scroll functionality. This would increase the area 
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available for display as it would remove the scroll bars typically provided at the extreme 
left or ri^t and top or bottom of a window. The gestures associated with the overloaded 
control element can determine both the direction and magnitude of the scrolling operation 
to be executed. The amount of scrolling can be proportional to the extent of the 2D 
5 gesture in the direction of the gesture. Further, the direction of scrolliug can be tiie same 
as ttie direction of the 2D gesture. For example, a short left goiug gesture made over the 
control element results in a small scroll to the left, and a long downward gesture made 
over the control element results in a large downward scroll. 



10 A fiirther control element could be made to be dependent on a combination of gesture and 
keyboard, or other input device, entry in order to execute some or all ftmctions. For 
example a control element could be used to close down or reset a device. In order to 
provide a failsafe mechanism. The ftmction associated with the gesture is not executed 
unless a user is also pressing a specific key, or key combination, on the devices keyboard 

1 5 at the same time. For example a soft reset of a device, could require a user to make a "x" 
gesture over the control element while also having the "CTRL" key depressed. Hence 
this would help to obviate incorrect gesture parsing, recognition or entry from 
accidentally causing harm. Further different combinations of keyboard keys and the same 
gesture could be used to cause different instructions to be executed. Hence^ keyboard 

20 entries and gestures could be combined to provide "short cuts" to selecting and executing 
different functions. 

A ftulher control element uses the semantic content of a gesture to ensure that the correct 
option or operation is carried out For example a control element could display a message 

25 and two options, for example "delete file" and the options "yes" and "no", M order to 
. execute the delete file operation, the user must make the correct type of mark which is 
conceptually related to the selected option. In this example, the user would make a "tick" 
mark to select yes, and a "cross" mark to select no. This would help prevent accidental 
selection of the uicorrect option as can happen currently when a user simply clicks on the 

30 wrong option by accident. The control element can fiirther be limited by requiring that 
the correct gesture be made over the corresponding region of the option of the control 
element. Hence, if a tick were made over the "no" option, then the conunand would not 
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be executed. Only making a tick over the region of the control element associated with 
the "yes" option would result in the command being executed TTiis provides a further 
safeguard. 

5 The methods and techniques of the current invention can be applied to user interfaces for 
many electrical devices, for exan^le to support interaction for Databoards, public 
information kiosks, small devices, such as wearable devices and control dashboards for 
augmented and virtual reaUty interfeces. The keyboard aspect can be extended by the use 
of predictive text For example, the specific first letter of a word can be entered using a 
10 gesture and a further gesture is used to define the length of the word. Successive groups 
of letters are then tapped on, (as with the T9 dictionary), to generate a list of possibilities. 
Also it is possible to enter specific letters in order to refine to search. 

There are other applications and developnaents of the principles taught herein. For 
15 example, it has been found that usera can perceive controls with indirect gaze making the 
model useful in peripheral displays, adaptive systems and designing interaction for the 
visually impaired, such as people who lose all sight other than peripheral vision. 
Adaptive displays could also benefit fix>m the fireedom to place new items or reconfigure 
displays without upsetting the layout of controls. 

20 

Another property is, that elements sharing the same motion appear grouped together. 
This approach can be used to implement widely dispersed menu options on a display 
without the necessary overhead of bounding them in borders, as is usually required to 
suggest a group relationship. 

25 

Further control elements can be designed benefiting fi:om theories of perception. Such 
adaptations of the control elements will help to minimise, and govern the effects of, 
visual rivahy, by introducing 3D control elements and dynamic shading of control 
elements. 

30 

Generally, embodiments of the present mvention employ various processes mvolving data 
stored in or transferred through one or more computer systems. Embodiments of the 
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present invention also relate to an apparatus for performing these operations. This 
apparatus may be specially constructed for the required purposes, or it may be a general- 
purpose computer selectively activated or reconfigured by a computer program and/or 
data structure stored in the computer. The processes presented herein are not inherentiy 
5 related to any particular computer or other apparatus. In particular, various genoral- 
purpose machines may be used with programs written in accordance wifh the teachings 
herein, or it may be more convenient to construct a more specialized apparatus to perform 
the required method steps. 

10 In addition, embodiments of tiie present invention relate to computer readable media or 
computer program products that include program instructions and/or data (including data 
structures) for performing various computer-implemented operations. Examples of 
computer-readable media include, but are not limited to, magnetic media such as hard 
disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; ihagneto- 

1 5 optical media; semiconductor memory devices, and hardware devices that are specially 
configured to store and perform program instructions, such as read-only memory devices 
(ROM) and random access memory (RAM). The data and program instructions of this 
invention may also be embodied on a carrier wave or other transport medium. Examples 
of program instructions include both machine code, such as produced by a compiler, and 

20 files containing higher level code that may be executed by the computer using an 
mterpreter. 

Alihough the above has generally described the present invention according to specific 
processes and apparatus, the present invention has a broad range of applicability. In 
25 particular, aspects of the present invention is not limited to any particular kind of 
electronic device. One of ordinary skill in die art would recognize other variants, 
modifications and alternatives in light of the foregoing discussion. 

It will also be appreciated that the invention is not limited to the specific combinations of 
30 structural features, data processing operations, data structures or sequences of method 
steps described and that, unless the context requires otherwise, the foregoing can be 
altered, varied and modified For example different combinations of features can be used 
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and features described wilii reference to one embodiment can be combined with other 
features described with reference to other embodiments. Similarly the sequence of the 
methods step can be altered and various actions can be combined into a single method 
step and some methods steps can be carried out as a plurality of individual steps. Also 
5 some of the features are schematically illustrated separately, or as comprising particular 
combinations of features, for the sake of clarity of explanation only and various of the 
features can be combined or integrated together. 

It will be appreciated that the speciJBc embodiments described above are cited by way of 
1 0 example, and that the present invention is not limited to what has been particularly shown 
and described hereinabove. Rather, the scope of the present invention mcludes both 
combinations and subcombinations of the varions features described hereinabove, as well 
as variations and modifications thereof which would occur to persons skilled in the art 
upon reading the foregoing description. 
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CLAIMS: 



1 . A user interface for a display of an electronic device, the user interface 
including: 

5 a background layer for displaying an interface; and 

at least a frst animated control element overlaid on the back ground layer, 
wherein the control element has a plurality of functions associated witii it and each of said 
functions being executable by making a 2D gesture associated with a one of said plurality 
of functions in a region of the user interface associated with the control elemmt 

10 

2. A user interface as claimed in claim 1, wherein the control element moves over a 
region of the display. 

3. A user interfece as claimed in claim 1 or claim 2, wherein the control elmient is- 
15 an icon- 

4. A user interface as claimed in claim 1 or 2, wherein the control element is an 
alphanumeric string. 

20 5. A user interface as claimed in claim 4, wherein the alpha numeric string is a 
word. 



6. A user interface as claimed in claim 5, wherein the word is polysyllabic and the 
each mdividual syllable is animated. 

25 

7. A user interface as claimed in claims 1 or claim 2, wherein the control element is 
a button. 

8. A user interface as claimed m claim 7, wherein the button bears and indicia 
30 indicating a menu of functions associated with the button and wherein making the 2D 

gesture executes a function from the menu. 
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9. A user interface as claimed in any preceding claim, wherein a help function is 
associated with the control element and wherein making a help 2D gesture causes help 
information relating to the functions associated with the control element to be displayed 
in the user interface. 

5 

10. A user inter&ce as claimed in claim 9» wherein the help 2D gesture has the shape 
substantially of a question made. 

11. A user interface as claimed in any preceding claim^ wherein tiie control element 
10 is visually opaque. 

12. A user interface as claimed in any of claims 1 to 10, wherein the control element 
is visually transparent. 

15 13. A user interface as claimed in claim 12, wherein the control element has a 
transparency of less than substantially 30%. 

14. A user interface as claimed in any preceding claim, wherein the user interfece 
includes a plurality of animated control elements. 

20 

15. A user interface as claimed in claim 14, wherein flie first control element is of a 
first type and a second of the plurality of control elements is of a second type, which is 
different to the first type. 

25 1 6. A user interface as claimed in claim 14 or 1 5, wherein the plurality of control 
elements between them provide a keyboard. 

17. A user interface as claimed in claim 16, wherein the keyboard has a standard 
layout 

30 

18. A user interface as clainied in claim 16 or 17 wherein the keyboard provides all 
of the characters in an alphabet of a language. 
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19. A user interface as claimed in any of claims 16 to 18, wherein at least one of ttie 
control elements is associated with a plurality of characters and each of the plurahty of 
characters has a respective 2D gesture associated therewith for causmg the character to be 
displayed on the background layer. 

5 

20. A user interface as claimed in any preceding claim wherein the control element 
has a 2D gesture associated with it for carrying out a formatting function on a character 
associated with the control element 

10 21- A user interface as claimed in any of claims 1 to 15, wherein at least one control 
elements is associated with a plurality of media player functions and each of the media 
player functions has a respective 2D gesture associated therewith for causing Ihe media 
player function to be executed. 

15 22. A user interface as claimed in any preceding claim, wherein the control element 
is animated so as to appear like a three dimensional entity. 

23. A user interface as claimed in any preceding claim, wherein the control element 
is animated so as to be more readily noticeable by peripheral vision. 

20 

24. A user interface as claimed in claim 23, wherein the control element has an axis 
along which it is animated. 

25. A user interface as claimed in claim 24, wherein the control elements animation 
25 comprises variable thickness bars scrolling along the axis. 

26. An electronic device having a user interface, the electronic device including: 
a display device; 

a data processing device; and 
30 a memory storing instructions executable by the data processing device to 

display the user interface on the display, wherein the user interface is as claimed in any 
preceding clainL 
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27. A device as claimed in claim 26, wherein the display is a touch sensitive display. 



28. A device as claimed in claim 26 or 27, wherein the device fiirliier includes a 
pointer device for making a 2D gesture on the user interface. 

5 

29. A device as claimed in any of claims 26 to 28, wherein die device is a handheld 
device. 

30. A device as claimed in any of claims 26 to 29, wherein the device is a wireless 
10 telecommunications device. 

31. A device as claimed in claim 30, wherein the device is a cellular 
telecommunications device. 

1 5 32. A computer implemented method for providing a user interface for a display of 
an electronic device, comprising: 

displaying an interface as a background layer; 

displaying an animated control element associated with a plurahty of functions 
over the background layer, 
20 detecting a 2D gesture made over a region of the user interface associated with 

the control element; and 

executing a one of the plurality of functions which is associated with the 2D 

gesture. 

25 33. A method as claimed in claim 32, wherein a plurality of animated control 
elements are displayed. 

34. A method as claimed in claim 32 or 33, wherein the animated control elements 
are transparent 

30 

35. A method as claimed in any of claims 32 to 34 and wherein detecting the 2D 
gesture further comprises a gesture engine parsing the 2D gesture and generating a 
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keyboard event corresponding to the 2D gesture. 
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34. A method as claimed in any of claims 32 to 35, and further comprising 
determining a location within the display of the 2D gesture and determining whether a 

5 control element is associated with the location. 

35. A me&od as claimed in any of claims 32 to 35, and further comprising: 
determining whether a gesture is intended to activate a control element and if not flien 
determining a function of the background layer to execute. 

10 

36. A method as claimed in claim 32, wherein tiie 2D gesture is a help 2D gesture 
and the fimction associated with the 2D gesture is a help function which displays 
information relating to the control element 

15 37. A method as claimed in claim 36, wherein the information relating to the control' 
element includes a graphical indication of the 2D gestures associated with the control 
element and/or text explaining the functions associated with the 2D control element 

38. A method as claimed in claim 32, wherein the control element is associated with 
20 a menu of functions and wherein the 2D gesture causes a one of the functions jfrom the 

menu of functions to be executed. 

39. A method as claimed in claim 33 wherein fhc plurality of control elements 
between them provide a key board and wherein the 2D gesture causes a character selected 

25 from the keyboard to be displayed on the background layer. 

40. A method as claimed in any of claims 32 to 39 wherein the control element is a 
character string. 

30 41. A method as claimed in claim 40, wherein the character string is a word. 



42. A metiiod as claimed in claim 41, wherein the word is a polysyllabic word and 
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each syllable of the word is separately animated 

43 • Computer program code executable by a data processing device to provide the 
user interface of any of clauns 1 to 25 or the computing device of any of claims 26 to 31 
or the method of any of claims 32 to 40. 

44. A computer program product comprising a computer readable medium bearing 
computer program code as claimed in claim 43, 



wo 2004/111816 



1/11 



PCT/GB2004/002538 




wo 2004/111816 



2/11 



PCT/GB2004/002S38 




132 



FIG. 2 





wo 2004/111816 



3/11 



PCT/GB2004/002538 



FIG. 4 

222. 



1/0 Stiream 



224 



OS 



226 



Gesture Eng 



228 



App 



GUI 



Event 



Overload 



Control 



210 



230 
-232 
234 



-236 




206 



208- 
212 



■214 



wo 2004/111816 



4/11 



PCT/GB2004/0D2S38 



FIG. 5 



262 




250 



I 



Initialize control 


256 


elements 





Display GUI and 
control elements 



258 



•260 



Detect 
gesture 

IT 



Execute operation 
associated with gesture 



264 



c 



266 



Done 



wo 2004/111816 



5/11 



PCT/GB2004/002538 





wo 2004/111816 



6/11 



PCT/GB2004/002538 




wo 2004/111816 



7/11 



PCT/GB2004/002538 



FIG. 8 



302 



c 



Begin 

31 



/ 



304 



300 



Gesture engine intercepts gesture 
inputted by pointing device 



I 



306 



Determine iceyboard event 
associated with gesture 



I 



System Message Event Handler 
intercepts keyboard events, pointer 
event and pointer co-ordinates 

310 



Pointer event in 
'a control element region ST 
movement within 
time out? 



318 



312 



Determine which control 
element region 



314 



320 



n 



Control element 
region & keyboard event 
associated with 
command? 



322 



Execute command 



308 



1 



System call 
passing event 
to underlying 

layer event 
handier 



Event handled 
by event 
handler for 
underlying 
layer 



T 

Done ) 



wo 2004/111816 



8/11 



PCT/GB2004/002538 




wo 2004/111816 



9/11 



PCT/GB2004/002538 




360 



FIG. IIA FIG. IIB FIG. IIC 



wo 2004/111816 



PCT/GB2004/002S38 



10/11 







Rewind Pause Play Stop Forward 



384 



380 



382 



FIG. 12 



Play 



Rewind 




FIG. 13 



VfO 2004/111816 



11/11 



PCT/GB2004/002538 



FIG. 14 



408 



402 



Begin 




? Gesture detected 
over control element 



Display 2D gestures 
corresponding and 
functions 




400 



404 



405 



406 




410 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 



1^ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 



p BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 



